Imaging device providing capture location guidance

ABSTRACT

An image capture device, comprising an image sensor, an image display, a data processing system; a storage memory and a program memory storing instructions configured to cause the data processing system to provide guidance for capturing images at a plurality of locations, by displaying guidance information for capturing a first digital image at a first location, using the image sensor to capture a first digital image, storing the first digital image in the storage memory, using the data processing system to analyze the pixel data of the first digital image and determining based on such analysis a second possible image capture location, wherein the second possible image capture location is selected from a plurality of different possible locations, displaying guidance information for a capturing a second scene at the second location, using the image sensor to capture a second digital image; and storing the second digital image in the storage memory.

CROSS REFERENCE TO RELATED APPLICATIONS

Reference is made to commonly-assigned, co-pending U.S. patentapplication Ser. No. 13/168,003, filed Jun. 24, 2011, entitled “Guidancefor Image Capture at different Locations”, by Timothy L. Nichols, et al.and U.S. patent application Ser. No. 13/168,027, filed Jun. 24, 2011,entitled “Photo Product Using Images from Different Locations” byTimothy L. Nichols, et al. the disclosures of which are incorporatedherein.

Reference is made to commonly assigned, co-pending U.S. patentapplication Ser. No. 12/693,621 by Dhiraj Joshi, et al. filed on Jan.26, 2010, entitled “On-location recommendation for photo composition”,U.S. patent application Ser. No. 12/692,815 by Jiebo Lou filed on Jan.25, 2010, entitled “Recommending places to visit”, U.S. patentapplication Ser. No. 12/914,310 by Tomi Lahcanski et al. filed on Oct.28, 2010, entitled “Method of locating nearby picture hotspots”, U.S.patent application Ser. No. 12/914,266 by Tomi Lahcanski et al. filed onOct. 28, 2010 and entitled “Method of locating nearby picture hotspotsand U.S. patent application Ser. No. 12/914,294 by Tomi Lahcanski et al.filed on Oct. 28, 2010, entitled “Organizing nearby picture hotspots”.

FIELD OF THE INVENTION

The present invention relates to image capture devices, and moreparticularly relates to image capture devices which provide guidance forcapturing images at different locations.

BACKGROUND OF THE INVENTION

Mobile phones, tablet computers, networked cameras, and other portabledevices incorporating camera modules and network connections to theInternet have opened up opportunities for new and exciting gaming,entertainment, and structured learning experiences. This technology iscurrently used to create geocache treasure hunt games and photo-basedscavenger hunt games. It is also used to enable museum tours as well astours of historic areas and other tourist attractions.

However, these experiences are relatively static. Typically, the game orexperience is designed once and played many times in a similar manner byall the users. In some cases, these games or experiences are provided,or modified, based on the location of the user. For instance, theGeocache Navigator, from Trimble Navigation Limited, Sunnyvale, Calif.is an application (APP) for a Smartphone which uses the phone's GPS andinterne connections to access live information directly fromgeocaching.com. This enables a user to locate geocache challenges whichare closest to their current location.

It is known to provide preference-aware location-based services, asdescribed in the paper titled “Toward context and preference-awarelocation-based services” authored by Mokbel et al. Such systems tailortheir services based on the preference and context of each customer. Forexample, in a restaurant finder application, the system can use thedietary restrictions, price range, other user ratings, current traffic,and current waiting time to recommend nearby restaurants to thecustomer, rather than recommending all of the closest restaurants.

Photography is often used to record and share experiences, such asvacation trips, family outings, or seasonal events. Still and videoimages of such experiences can be captured using image capture devicessuch as camera phones, digital cameras, and camcorders. The digitalimages captured by these image capture devices can be shared by e-mailand uploaded to web sites such as Facebook and Flickr, where they can beviewed by friends. The uploaded images can be printed using photoservice providers, such as the Kodak Gallery at www.kodakgallery.com.Users can order photo products, such as photo books and collages, whichutilize uploaded digital images.

It is known to produce enhanced photo products by combining imagescaptured with an image capture device and professionally produceddigital content, as described in commonly-assigned U.S. patentapplication Ser. No. 11/626,471 (published as 20080174676), “Producingenhanced photographic products from images captured at known events” toSquilla, et al, incorporated herein by reference. The system includes adatabase for storing custom content for a plurality of events. Thesystem also includes a digital image capture device that stores adigital image and information defining the date/time and geographiclocation of the digital image. A service provider automaticallydetermines if the timestamp and the geographic information correspondsto events stored in the custom content database. A processor produces anenhanced photographic product including the captured digital image andcustom content corresponding to the timestamp and location of thecaptured digital image.

It is known to use image recognition techniques to produce aphotocollage from a plurality of images, as described incommonly-assigned U.S. Pat. No. 6,389,181 “Photocollage generation andmodification using image recognition” to Shaffer et al, incorporatedherein by reference. The system sorts digital records associated with aplurality of images, by culling or grouping to categorize the recordsaccording to an event, person, or chronology, in order to automaticallycompose a photocollage.

What is needed is an image capture device, such as a smart phone, whichprovides guidance for capturing images at different locations. The imagecapture device should preferably provide a dynamic, compelling,photo-based experience responsive to the user, situation, andconditions.

SUMMARY OF THE INVENTION

In accordance with the invention, there is provided an image capturedevice, comprising:

a) an image sensor for capturing a digital image of a scene;

b) an image display;

c) a data processing system;

d) a storage memory for storing captured images; and

e) a program memory communicatively connected to the data processingsystem and storing instructions configured to cause the data processingsystem to control elements a)-e) to provide guidance for capturingimages at a plurality of locations, by:

-   -   i) displaying, on the image display, guidance information for        capturing a first digital image at a first location;    -   ii) using the image sensor to capture a first digital image;    -   iii) storing the first digital image in the storage memory    -   iv) using the data processing system to analyze the pixel data        of the first digital image and determining based on such        analysis a second possible image capture location, wherein the        second possible image capture location is selected from a        plurality of different possible locations;    -   v) displaying, on the image display, guidance information for        capturing a second scene at the second location;    -   vi) using the image sensor to capture a second digital image;        and    -   vii) storing the second digital image in the storage memory.

It is an advantage of the present invention to provide an image capturedevice which provides an appropriate experience for capturing images atdifferent locations.

It is a further advantage of the present invention to provide an imagecapture device which provides guidance to a user in order to adapt theexperience for different users, situations, or conditions.

It is a feature of the invention that the guidance is provided afteranalyzing the pixel data of a captured digital image, in order todetermine a second possible image capture location based on the user,situation, or condition.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a digital imaging system in accordance withan embodiment of the present invention;

FIG. 2 is a block diagram of a camera phone used in the digital imagingsystem of FIG. 1;

FIG. 3 is a high level flow diagram depicting steps for providingguidance for image capture at different locations;

FIG. 4A and FIG. 4B depict two different examples of guidance for imagecapture at different locations based on an analysis of the previousimage received;

FIG. 5 is a high level flow diagram depicting steps for generating aphoto product from images captured at different locations;

FIG. 6A-6C depict pages of a first photo product which includes selectedimages positioned in the photo product in association with prestoredinformation;

FIG. 7A-7C depict pages of a second photo product which includesselected images positioned in the photo product in association withprestored information; and

FIG. 8A and FIG. 8B depict two different example photo products createdwith images received from users which were captured at the same locationand utilize different prestored information.

It is to be understood that the attached drawings are for purposes ofillustrating the concepts of the invention and may not be to scale.

DETAILED DESCRIPTION OF THE INVENTION

In the following description, some embodiments of the present inventionwill be described in terms that would ordinarily be implemented as asoftware program. Those skilled in the art will readily recognize thatthe equivalent of such software can also be constructed in hardware.Because image manipulation algorithms and systems are well known, thepresent description will be directed in particular to algorithms andsystems forming part of, or cooperating more directly with, the systemand method in accordance with the present invention. Other aspects ofsuch algorithms and systems, and hardware or software for producing andotherwise processing the image signals involved therewith, notspecifically shown or described herein, can be selected from suchsystems, algorithms, components and elements known in the art. Given thesystem as described according to the invention in the followingmaterials, software not specifically shown, suggested or describedherein that is useful for implementation of the invention isconventional and within the ordinary skill in such arts.

Still further, as used herein, a computer program for performing themethod of the present invention can be stored in a non-transitorycomputer readable storage medium, which can include, for example;magnetic storage media such as a magnetic disk (e.g., a hard drive or afloppy disk) or magnetic tape; optical storage media such as an opticaldisc, optical tape, or machine readable bar code; solid state electronicstorage devices such as random access memory (RAM), or read only memory(ROM); or any other physical device or medium employed to store acomputer program having instructions for controlling one or morecomputers to practice the method according to the present invention.

The invention is inclusive of combinations of the embodiments describedherein. References to “a particular embodiment” and the like refer tofeatures that are present in at least one embodiment of the invention.Separate references to “an embodiment” or “particular embodiments” orthe like do not necessarily refer to the same embodiment or embodiments;however, such embodiments are not mutually exclusive, unless soindicated or as are readily apparent to one of skill in the art. The useof singular or plural in referring to the “method” or “methods” and thelike is not limiting. It should be noted that, unless otherwiseexplicitly noted or required by context, the word “or” is used in thisdisclosure in a non-exclusive sense.

Because wireless image capture devices and systems, such as cameraphones connected via cellular telephone systems to service providersusing the Internet are well known, the present description will bedirected in particular to elements forming part of, or cooperating moredirectly with, the method and apparatus in accordance with the presentinvention. Elements not specifically shown or described herein areselected from those known in the art. Certain aspects of the embodimentsto be described are provided in software. Given the system as shown anddescribed according to the invention in the following materials,software not specifically shown, described or suggested herein that isuseful for implementation of the invention is conventional and withinthe ordinary skill in such arts.

The following description of image capture devices and imaging systemswill be familiar to one skilled in the art. It will be obvious thatthere are many variations of this embodiment that are possible and areselected to reduce the cost, add features, or improve the performance ofthese devices and systems. The present invention is illustrated by wayof example and not a limitation in the accompanying figures.

Referring to FIG. 1, there is illustrated a system 214 for capturingdigital images along with location and time information, and using theimages and information to provide customized photo products. As usedherein the term digital image includes both digital still images anddigital video images. A first camera phone 300A, located at a firstlocation A, and a second camera phone 300B, located at a second locationB, can communicate using a cellular provider network 240. The cellularprovider network 240 provides both voice and data communications usingtransmission devices located at cell towers throughout a region. Thecellular provider network 240 is coupled to a communication network 250,such as the Internet. It will be understood that system 214 typicallyincludes many other camera phones, in addition to camera phone 300A andcamera phone 300B. It will be understood that the system 214 can includemultiple cellular provider networks 240, for example networks providedby companies such as Verizon, AT&T, and Sprint, which can be coupled tothe communication network 250.

The communications network 250 enables communication with a serviceprovider 280. Service provider 280 includes a web server 282 forinterfacing with communications network 250. In addition to interfacingto communications network 250, web server 282 transfers information to acomputer system 286 which manages images and information associated withvarious customers and with image content associated with differentlocations and events. It will be understood that the system 214 caninclude a plurality of service providers 280, which provide differentservices and can support different regions of the world.

The computer system 286 includes an account manager 284, which runssoftware to permit the creation and management of individual customerphoto imaging accounts and to also permit the creation and management ofcollections of custom content images, such as professional images, andother content associated with various events and locations. The customerimages and associated information are stored in a customer database 288.The customer account information can include personal information suchas name and address, billing information such as credit cardinformation, and authorization information that controls access to thecustomer's images by third parties. The professional images and othercustom content associated with the supported events and locations arestored in custom content database 290.

Thus, the customer database 288 stores customer image files and relatedmetadata, such as location and time information which identifies thelocation at which the image was captured, and the time of capture. Thecustom content database 290 stores custom content, such asprofessionally captured images and other information, such as captions,titles, text, graphics, templates, and related metadata. For example,the custom content database 290 can store images and other informationrelated to particular vacation destinations (e.g. Washington D.C., NewYork City, Cape May N.J.) and particular events (Rose Bowl Parade,Professional Sports events, Major Concerts.). The custom contentdatabase 290 includes an index providing location or event data such asthe GPS coordinate boundaries of locations, object identifying featurepoints, object identifying color profiles, or the time boundaries ofevents, so that locations (such as Cape May, or Yellowstone NationalPark) and events (such as the Rose Bowl Parade or the Rochester LilacFestival) can be identified.

The custom content database 290 also stores guidance information, whichis used to provide guidance to a user concerning what images should becaptured by a user in a general location. In some embodiments, theguidance information provides locations which are likely to beconsidered to be good “photo spots” by the particular user of one of thecamera phones 300A, 300B. In some embodiments, the guidance informationincludes at least one image related to the suggested location. Forexample, the guidance can include a photo of a particular object, alongwith a text message that provides a general direction, or other clues,for locating the object. In some embodiments, the guidance can alsoinclude text or graphics which instruct the user to capture an image oftheir group near the object, and to email the image to the serviceprovider.

In some embodiments, guidance for capturing images at differentlocations is provided in a manner so as to dynamically alter thephoto-based experience responsive to input received during theexperience. In this way, the experience adapts to a particular user'ssituation and conditions. For example, a photo submitted at one point inthe experience can indicate that the user is accompanied by children.This can result in future experience objectives being more suitable to ayounger audience. In another example, input received from the user canindicate that it is raining or snowing. In this condition, futureexperience objectives can be tailored to indoor venues.

The computer system 286 includes a processor 292, which is used toanalyze the pixel data of some of the customer images which are uploadedand stored in the customer database 288. For example, in someembodiments the processor 292 can analyze the pixel data in order todetect faces in one or more customer images using a variety of knownface detection algorithms. Such algorithms are described, for example,in a paper titled “Comparative Testing of Face Detection Algorithms” byDegtyarev et al., which is available fromhttp://lda.tsu.tula.ru/papers/degtyarev-2010-icisp-ctfd.pdf and isincorporated herein by reference. In some embodiments, the facedetection algorithm determines the number of faces that can be detectedin an image, in order to determine how many people are depicted in theimage. In some embodiments, the face detection algorithm determines theapproximate ages of the people whose faces have been detected. It willbe understood that the term approximate age, as used herein, relates tocategorizing one or more faces into broad, age-related categories. Theseapproximate age categories can include, for example, babies, youngchildren, teens, younger adults, and older adults (i.e. seniorcitizens).

In some embodiments, the processor 292 in the computer system 286 cananalyze the pixel data of some of the customer images in order todetermine whether one or more landmarks are depicted in the images. Suchimage recognition algorithms are used, for example, in the GoogleGoggles Application (APP) for the Android mobile platform, which isavailable from Google, Mountain View, Calif.

In some embodiments, the processor 292 in the computer system 286creates the information needed to provide a unique photo product for aparticular user of one of the mobile phones 300A, 300B by incorporatingimages captured during the user's photo-based experience with prestoredinformation, such as professional images and textual descriptions. Thisenables a photo product to be automatically created by placing thecaptured images in predetermined locations in the photo product, so thatthey are associated with the prestored information. For example, a firstimage captured near the Lincoln Memorial in Washington D.C. can beassociated with prestored information which describes the presidency ofAbraham Lincoln and provides professional photographs of the LincolnMemorial or an image related to his Gettysburg Address speech. A secondimage, captured near the White House, can be associated with prestoredinformation that describes or depicts the current president or theconstruction of the White House.

In some embodiments, the processor 292 in the computer system 286modifies the appearance of one or more of the captured digital images,so that it has a more suitable appearance when incorporated into thephoto product. In some embodiments, faces in the captured digital imagecan be detected, and the processor 292 can crop the digital image toenlarge the size of the faces and remove some of the distractingbackground surrounding the face.

In some embodiments, captured digital images can be processed by theprocessor 292 to provide a different image appearance. For example,captured digital images can be processed so that the newly capturedimages appear to be older photographs, such as daguerreotypes, so thatthey have a more suitable appearance when positioned in a photo productin association with an image related to the Gettysburg Address. Asanother example, the captured digital images can be processed to providean image having a different color tint, contrast, or external shape, sothat it has a more suitable appearance when positioned in a photoproduct as part of an advertisement for a product or service. As anotherexample, the captured digital images can be processed to provide acartoon effect or a coloring book effect so that they have a moresuitable appearance when positioned in a children's photo product inassociation with prestored cartoons or as part of a page which providesa “coloring book” for a child.

In some embodiments, captured digital images can be processed by theprocessor 292 to provide a different image appearance in response to theimage content of the captured image. For example, the processor 292 candetermine the location of multiple faces within the image andautomatically crop the captured digital image using different aspectratios for different captured images in order to produce a more suitableappearance in the photo product.

In some embodiments, the captured digital images can be processed by theprocessor 292 to provide a different image appearance in response to thelocation where the image was captured. For example, the processor 292can provide a “cartoon” effect for images captured in a particularlocation, such as images captured in a particular park or playground.

In some embodiments, the captured digital images can be processed by theprocessor 292 to provide a different image appearance in response toboth the image content of the captured image and the location where theimage was captured. For example, the processor 292 can provide acolor-based object extraction algorithm (e.g. “green screen” effect”) onimages captured in a particular location when the processor 292 candetermine that a background area of the captured image is apredetermined color (e.g. green). The communications network 250 enablescommunication with a fulfillment provider 270. The fulfillment provider270 produces and distributes enhanced photo products. The fulfillmentprovider 270 includes a fulfillment web server 272, and a fulfillmentcomputer system 276 that further includes a commerce manager 274 and afulfillment manager 275. Fulfillment requests received from serviceprovider 280 are handled by commerce manager 274 initially beforehanding the requests off to fulfillment manager 275. Fulfillment manager275 determines which equipment is used to fulfill the ordered good(s) orservices such as a digital printer 278 or a DVD writer 279. The digitalprinter 278 represents a range of color hardcopy printers that canproduce various photo products, including prints and photo albums. Thehardcopy prints can be of various sizes, including “poster prints”, andcan be sold in frames. The DVD writer 279 can produce CDs or DVDs, forexample PictureCDs, having digital still and video images andapplication software for using the digital images.

After fulfillment, the photo products are provided to the user of thecamera phones 300A, 300B, or to a recipient designated by the user ofthe camera phone 300 sA, 300B. In some embodiments, the photo productsare provided using a transportation vehicle 268. In other embodiments,the photo products are provided at a retail outlet, for pickup by theuser of the camera phones 300A, 300B, or by a designated recipient.

System 214 also includes one or more kiosk printers 224 whichcommunicate with the communication network 250 and service provider 280via a communication service provider (CSP) 222. This enables printedphoto products, created by the service provider 280 using digital imagescaptured by camera phones 300A, 300B, to be provided at retailestablishments.

System 214 also includes one or more customer computers 218 whichcommunicate with the communication network 250 and service provider 280via a communication service provider (CSP) 220. This enables photoproducts, created by the service provider 280 using digital imagescaptured by camera phones 300A, 300B, to be ordered at a later time bythe user of the camera phones 300A, 300B, or by others who areauthorized by the user of the camera phones 300A, 300B.

It will be understood that in some embodiments, a plurality of serviceproviders 280, fulfillment providers 270 or kiosk printers 224 can belocated at a plurality of different retail outlets. For example,fulfillment providers 270 can be located in a portion of a store whichis near a vacation spot or other attraction. In some embodiments, theuser of the camera phones 300A, 300B can be guided to the location of anearby fulfillment provider 270 in order to pick up a photo product thathas been produced using their captured digital images. In someembodiments, the user of the camera phones 300A, 300B receives the photoproduct at a discount, or free of charge, in order to encourage the userto enter the store where they will potentially purchase other items. Insome embodiments, the photo product includes advertising of merchantswhich are located near the location of the fulfillment provider 270.

In some embodiments, the service provider 280, or the fulfillmentprovider 270 can create examples of various photo products that can beprovided by the fulfillment provider 270, as described incommonly-assigned U.S. Pat. No. 6,915,723 entitled “Method For ProvidingCustomized Photo Products Over A Network” by Parulski et al., thedisclosure of which is incorporated herein by reference. The examplescan be communicated to the camera phone 300 or the customer computer218, where the examples can be displayed to the user.

In some embodiments, the customer database 288 at the service provider280 includes information describing customer accounts for a plurality ofusers, including user billing information. The billing information caninclude a payment identifier for the user, such as a charge card number,expiration date, user billing address, or any other suitable identifier.In some embodiments, the customer database 288 also provides long-termstorage of the uploaded images for some or all of the users. In someembodiments, stored images are accessible (e.g., viewable) via theInternet by authorized users. Users can be authorized to view, print, orshare images as described in commonly-assigned U.S. Pat. No. 5,760,917,entitled “Image distribution method and system” to Sheridan, thedisclosure of which is incorporated herein by reference.

When a photo product is purchased by the user of the camera phones 300A,300B, the service provider account manager 284 can communicate with aremote financial institution (not shown) to verify that the paymentidentifier (e.g., credit card or debit card number) provided by thecustomer is valid, and to debit the account for the purchase.Alternatively, the price of the photo product can be added to the user'smonthly bill paid to the service provider 280 or to their mobile phoneoperator.

It will be understood that in some embodiments, the functions of theservice provider 280 and the fulfillment provider 270 can be combined,for example, by using a common web server for both web server 282 andweb server 272 or by combining the functions of the account manager 284,the commerce manager 274, and the fulfillment manager 275. It will beunderstood that in some embodiments, the customer database 288 or thecustom content database 290 can be distributed over several computers atthe same physical site, or at different sites.

FIG. 2 depicts a block diagram of a camera phone 300 used in the digitalphotography system of FIG. 1. The camera phone 300 can send and receiveemail messages and text messages which include images. It will beunderstood that other types of image capture devices, such as a wirelessdigital camera, can be used in the system described in reference toFIG. 1. The camera phone 300 or other type of image capture device canalso include other functions, including, but not limited to, thefunctions of a digital music player (e.g. an MP3 player), a GPSreceiver, or a programmable digital assistant (PDA).

The camera phone 300 is a portable battery operated device, small enoughto be easily handheld by a user when capturing and reviewing images. Thecamera phone 300 includes a lens 304 which focuses light from a scene(not shown) onto an image sensor array 314 of a CMOS image sensor 310.The image sensor array 314 can provide color image information using thewell-known Bayer color filter pattern. The image sensor array 314 iscontrolled by timing generator 312, which also controls a flash 302 inorder to illuminate the scene when the ambient illumination is low. Theimage sensor array 314 can have, for example, 2560 columns×1920 rows ofpixels.

In some embodiments, the digital camera phone 300 can also store videoclips by summing multiple pixels of the image sensor array 314 together(e.g. summing pixels of the same color within each 4 column×4 row areaof the image sensor array 314) to create a lower resolution video imageframe. The video image frames are read from the image sensor array 314at regular intervals, for example using a 30 frame per second readoutrate.

The analog output signals from the image sensor array 314 are amplifiedand converted to digital data by the analog-to-digital (A/D) convertercircuit 316 on the CMOS image sensor 310. The digital data is stored ina DRAM buffer memory 318 and subsequently processed by a digitalprocessor 320 controlled by the firmware stored in firmware memory 328,which can be flash EPROM memory. The digital processor 320 includes areal-time clock 324, which keeps the date and time even when the digitalcamera phone 300 and digital processor 320 are in their low power state.The digital processor 320 produces digital images that are stored asdigital image files using image/data memory 330. The phrase “digitalimage” or “digital image file”, as used herein, refers to any digitalimage file, such as a digital still image or a digital video file.

The processed digital image files are stored in the image/data memory330, along with the date/time that the image was captured provided bythe real-time clock 324 and the location information provided by GPSreceiver 360. The image/data memory 330 can also be used to store otherinformation, such as phone numbers or appointments. In some embodiments,the camera phone 300 is a smart phone, and the digital processor 320uses a software stack, such as Android, which includes an operatingsystem, middleware, and applications. This permits a softwareapplication (“APP”) to be downloaded, stored in the firmware memory 328,and used to provide various functions.

In some embodiments, the digital processor 320 performs colorinterpolation followed by color and tone correction, in order to producerendered sRGB image data. In some embodiments, the digital processor 320can also provide various image sizes selected by the user. In someembodiments, rendered sRGB image data is then JPEG compressed and storedas a JPEG image file in the image/data memory 330. In some embodiments,the JPEG file uses the so-called “Exif” image format. This formatincludes an Exif application segment that stores particular imagemetadata using various TIFF tags. Separate TIFF tags are used to storethe date and time the picture was captured and the GPS co-ordinates, aswell as other camera settings such as the lens f/number.

In some embodiments, the digital processor 320 also creates alow-resolution “thumbnail” size image, which can be created as describedin commonly-assigned U.S. Pat. No. 5,164,831 entitled “Electronic StillCamera Providing Multi-Format Storage Of Full And Reduced ResolutionImages” to Kuchta, et al., the disclosure of which is hereinincorporated by reference. The thumbnail image can be stored in RAMmemory 322 and supplied to a color display 332, which can be, forexample, an active matrix LCD or organic light emitting diode (OLED).After images are captured, they can be quickly reviewed on the color LCDimage display 332 by using the thumbnail image data.

The graphical user interface displayed on the color display 332 iscontrolled by user controls 334. The graphical user interface enablesthe user to control the functions of the camera phone 300, for example,to capture still or video images, and to send or view text messages oremail messages. User controls 334 typically include some combination ofbuttons, rocker switches, or joysticks. In some embodiments, many of theuser controls 334 are provided by using a touch screen overlay on thecolor display 332. In other embodiments, the user controls 334 caninclude a means to receive input from the user or an external device viaa tethered, wireless, voice activated, visual or other interface. Inother embodiments, additional status displays or images displays can beused.

An audio codec 340 connected to the digital processor 320 receives anaudio signal from a microphone 342 and provides an audio signal to aspeaker 344. These components can be used both for telephoneconversations and to record and playback an audio track, along with avideo sequence or still image. The speaker 344 can also be used toinform the user of an incoming phone call. This can be done using astandard ring tone stored in firmware memory 328, or by using a customring-tone downloaded from the service provider 280. In addition, avibration device (not shown) can be used to provide a silent (e.g., nonaudible) notification of an incoming phone call, e-mail, or textmessage.

A dock interface 362 can be used to connect the camera phone 300 to adock/charger 364, which is connected to the customer computer 218. Thedock interface 362 can conform to, for example, the well-know USBinterface specification. Alternatively, the interface between thedigital camera phone 300 and customer computer 218, can be a wirelessinterface, such as the well-known Bluetooth wireless interface or thewell-know 802.11b wireless interface. The dock interface 362 can be usedto download image files (which include the date/time and GPScoordinates) from the image/data memory 330 to the customer computer218. The dock/charger 364 can also be used to recharge the batteries(not shown) in the digital camera phone 300.

The digital processor 320 is coupled to a wireless modem 350, whichenables the digital camera phone 300 to transmit and receive informationvia an RF channel 352. The wireless modem 350 communicates over a radiofrequency (e.g. wireless) link with the cellular provider network 240,which can utilize, for example, a CDMA network, a 3GSM, a 4 GSM network,or other wireless communication networks.

It will be understood that the functions of digital processor 320 can beprovided using a single programmable processor or by using multipleprogrammable processors, including one or more digital signal processor(DSP) devices. Alternatively, the digital processor 320 can be providedby custom circuitry (e.g., by one or more custom integrated circuits(ICs) designed specifically for use in camera phones), or by acombination of programmable processor(s) and custom circuits. It will beunderstood that connectors between the digital processor 320 from someor all of the various components shown in FIG. 2 can be made using acommon data bus. For example, in some embodiments the connection betweenthe digital processor 320, the DRAM buffer memory 318, the image/datamemory 330, and the firmware memory 328 can be made using a common databus.

FIG. 3 is a high level flow diagram depicting steps for providingguidance for image capture at different locations. In some embodiments,all of the steps are performed by the service provider 280 in FIG. 1. Inother embodiments, some or all of the steps are performed by the cameraphone 300 in FIG. 2. The guidance that is provided enables userexperiences where images are being captured and the pixel data of thecaptured image is analyzed so that the experience can be dynamicallymodified based on information determined as a result of the analysis ofthe captured images.

As a first example, determining that there are children playing the gamecan alter the difficulty of the game (to make it easier) or thelocations that the guidance suggests as the next scene to be captured(to be more appropriate for children). As a second example, determiningthat there is a large group of people in the image can cause theexperience to be dynamically modified so that the group is givendifferent tasks, at different locations, than would be the case withsingle individuals or couples. In some embodiments, the experience canbe dynamically altered based on ambient condition information, such asthe time of day (e.g. whether it is morning, afternoon, or evening) orthe weather conditions (e.g. whether it is snowing, rainy, or sunny). Insome embodiments, the ambient condition information includes geolocationinformation, such as GPS metadata.

In receive image step 400 of FIG. 3, a captured image is received. Insome embodiments, the image is received by a server, such as the webserver 282, over a communication network, such as communication network250. In some embodiments, the image and accompanying data (such as thedate and time of image capture, and the GPS location) is transmittedfrom a camera phone 300 over the communication network 250. In someembodiments, the image and accompanying data is transmitted inassociation with a text message, which can be transmitted using the MMS(Multimedia Messaging Service) protocol.

Upon receiving the message, the web server 282 can identify the customeror user. In some embodiments, the customer or user is identified by thetelephone number of the camera phone 300 that is transmitted with theMMS message. The image and accompanying data are then stored in thecustomer database 288.

In analyze image step 405 of FIG. 3, the pixel data of the receiveddigital image is analyzed. In some embodiments, the pixel data of thereceived digital image is analyzed by the processor 292 in the computersystem 286. The analysis uses one or more digital image analysistechniques in order to determine additional metadata from the pixel dataof the received image. These digital image analysis techniques caninclude, for example, semantic analysis, feature point identification,color map identification, facial identification, facial recognition, agerecognition, and color or light balance analysis.

In some embodiments, the digital image analysis is performed responsiveto other image metadata, such as geographic location data or time of daydata. For example, the digital image analysis can use a database relatedto landmarks at different locations, and the pixel data of the receivedimage can be analyzed to determine if any of the objects depicted in theimage are likely to be one of the landmarks in the vicinity of thegeographic location metadata associated with the received digital image.The newly determined metadata, for example the number of personsdepicted in the received image, or the approximate age of one of more ofthe persons depicted in the received image, can be stored in thecustomer database 288.

In some embodiments, the analysis of the image can permit the serviceprovider 280 to determine whether or not the user captured an imageconsistent with an intended objective provided by the service provider280, prior to receiving the captured digital image in receive image step400. In such embodiments, analyze image step 405 can determine that thereceived digital image is not consistent with the intended objectivesince it does not meet a predetermined criteria. For example, the pixeldata of the received image might not include a landmark that the userwas asked to find and photograph. In another example, the geographiclocation data associated with the image might not correspond to thelocation that the user was asked to find and photograph. In eitherexample, the service provider 280 can provide additional guidance to theuser, in order to provide additional instructions or “hints” that helpthe user locate the landmark or location.

In provide possible locations step 410, a plurality of possiblelocations is provided, so that a suitable next possible image capturelocation can be determined by selecting one of the plurality of possiblelocations. In some embodiments, the account manager 284 and the customerdatabase 288 in the computer system 286 are used to determine userspecific information related to the history of the user's interactionswith the system, as well as any previously captured or determinedinformation about the user's experience. For example, in a “treasurehunt” type scenario to be described later, the user may be known to betraveling a particular branch of a predefined hunt route. Further, itmay be known that the user has already completed three stages of thehunt and that previous stages have indicated the user was outside on abright sunny day.

In some embodiments, in provide possible locations step 410, the customcontent database 290 is accessed to determine the set of all possiblenext locations that could be sent to the user, given the user's history.

In final location test 415, a determination is made as to whether theexperience for this user should be concluded (yes to test 415) orwhether there is at least one additional image to be captured by theuser (no to test 415).

In some embodiments, this determination is made based on a user's knownposition on a predefined route. For example, when the user is near aparticular printing location, the experience for this user can beterminated and guidance can be provided to the user in order to instructthe user to pick up their free photo product at the nearby location.

In some embodiments, this determination can be made based on explicitinstructions the user conveyed in the most recent experienceinteraction. For example, when the experience begins, guidance can beprovided to the user to inform the user that the experience can be endedwhen the user sends a particular text message (e.g. “end”) from thecamera phone 300 to the service provider 280.

In some embodiments, this determination can be made based on the elapsedtime between the beginning of the experience interaction and the currenttime of day. For example, the experience can be automatically terminatedafter a predetermined time period (e.g. 30 minutes) has elapsed.

In some embodiments, this determination can be made based on ambientconditions, such as the current weather, the time of day, or safetyrelated ambient condition information. For example, the experience canbe automatically terminated if there is a severe weather storm in thearea, after the sun sets, or if a fire, crime, or other safety relatedincident occurs in the vicinity.

If there is at least one additional image to be captured by the user (noto final image test 415), then in determine next location step 420, oneof the plurality of next possible image capture locations is selected.In some embodiments, the computer system 286 in the service provider 280determines the next possible image capture location based on the resultof analyzing the pixel data of the received captured digital image inanalyze image step 405, such as whether there are any children depictedin the captured digital image. In some embodiments, ambient conditioninformation (such as whether it is a bright sunny day) is also used toautomatically determine the most appropriate next location from the setof possible next locations.

In provide guidance step 430, guidance is provided to the userconcerning the next possible image capture location that was determinedin determine next location step 420. In some embodiments, the computersystem 286 accesses the custom content database 290 to select theguidance appropriate to the next location selected in determine nextlocation step 420. In some embodiments, the guidance is then transmittedover communication network 250 to the user's camera phone 300. Theguidance will typically be in the form of image and text data, such asan MMS message, but can be of any format or type suitable fortransmission over the communication network 250.

In some embodiments, the guidance to the user can be provided by placinga phone call to the camera phone 300 for the particular user, using thephone number provided in the MMS message which included the captureddigital image. The phone call can provide one of a plurality ofprerecorded messages which provides the guidance for the next locationwhich was determined in determine next location step 420. For example,the prerecorded message can be recorded by an actor, pretending to be ahistoric figure associated with the theme of the user experience. Theprerecorded message can describe, in a historic context, the next sceneto be captured at the next location.

In some embodiments, the guidance to the user can include dynamicallyconstructed images using the user's submitted image in combination withprestored information. In some embodiments, the user's submitted imagecan be modified and composited with prestored information. For example,the processor 292 in the computer system 286 can process the receivedcaptured image in order to crop out a face of a person depicted in theimage, convert the face from a color to a monochrome image, andcomposite the image of the face into one of a plurality of prestorednewspaper templates, so that the newly captured images appears to be aphotograph in a historic newspaper related to a historic site whichserves as the theme of the experience. The newspaper text can describethe next scene to be captured at the next location which was determinedin determine next location step 420. The newspaper text can be modifiedbased on text entered by the user of the camera phone 300. For example,the headline of the newspaper can read “Matt hunts the ghost of SamPatch”, or alternately “Troop 79 hunts the ghost of Sam Patch” if theuser entered “Matt” or “Troop 79” as the individual or group name, inresponse to earlier guidance provided to the user of the camera phone300.

In some embodiments, the service provider 280 provides experiencespecific content advertisements, or coupons specific to the user'sexperience, over the communication network 250 to the camera phone 300.These advertisements can be transmitted over communication network 250as independent messages, or bundled into the response generated byprovide guidance step 430.

In some embodiments, the user's submitted image can be modified andcomposited with prestored information in order to create theadvertisements or coupons. In some embodiments, a particularadvertisement is selected from a plurality of possible advertisementsbased on various criteria. The criteria can include, for example, theapproximate age of one or more of the persons depicted in the captureddigital image. For example, if the captured digital image includes oneor more children, the particular advertisement can be for anage-appropriate book or toy related to the theme of the experience. Thecriteria can also include, for example, weather related information suchas the current temperature. For example, on warm days the advertisementcan provide an offer related to a discount on an ice cream cone at afirst nearby merchant, and on cold days the advertisement can provide anoffer related to a discount on a hot drink at second nearby merchant. Insome embodiments, the coupons can be for a limited time period, based onthe date and time ambient condition information. In some embodiments,the coupons can customized so that they can only be used by theparticular user of the camera phone 300. This can be done, for example,by including one of the digital images captured by the user, as part ofthe coupon.

In some embodiments, the user of the camera phone 300 can send a messagewhich rejects the next location determined in determine next locationstep 420. In response, the service provider 280 can determine analternative next location and transmit guidance to the user whichincludes information concerning the alternate second location. Forexample, the user can decide to reject the next location based on thedifficulty in finding the next location.

If there are no more images to be captured by the user (yes to finallocation test 415), then one or more photo products is created increated photo product block 440. The photo products use a combination ofthe images received from the user during the experience as well asprestored content. In some embodiments, the prestored content isselected based on analyzing the pixel data of one or more captureddigital images in analyze image step 405. The photo products can becreated as will be described in more detail in reference to FIG. 5.

In some embodiments, some or all of the steps described in reference toFIG. 3 can be provided by an image capture device, such as camera phone300. In some embodiments, the camera phone 300 is a smart phone, and theservice provider 280 provides a downloadable software application(“APP”) over the communication network 250 to the camera phone 300. Thecamera phone 300 is one example of an image capture device, and includesan image sensor array 314 for capturing a digital image of a scene, acolor display 332, a digital processor 320 which serves as a dataprocessing system, image/data memory 330 which serves as a storagememory for storing captured images; and firmware memory 328 which servesas a program memory. The firmware memory 328 is communicativelyconnected to digital processor 320.

In this example, the instructions provided in the APP can control thedigital processor 320 in order to display, on the color display 332,guidance information for capturing a first digital image at a firstlocation; and enable the camera phone 300 to capture a first digitalimage using the image sensor array 314 and store the first digital imagein image/data memory 330. The instructions provided in the APP can thencontrol the digital processor 320 in the camera phone 330 to analyze thepixel data of the first digital image and to determine a second possibleimage capture location from a plurality of different possible locationsprovided by the APP.

The instructions provided in the APP can then control the digitalprocessor 320 in the camera phone 300 to display, on the color display332, guidance information for capturing a second scene at the selectedsecond location, and enable the camera phone 300 to capture a seconddigital image using the image sensor array 314, and store the seconddigital image in image/data memory 330.

In some embodiments, the instructions provided in the APP can cause thedigital processor 320 to provide guidance to the user concerning analternate second location responsive to an input provided, using a userinterface such as the user controls 334, rejecting the second location.

In some embodiments, the first and second captured images aretransmitted to the service provider 280 over the wireless modem 350, sothat the service provider 280 can create one or more photo productsusing the first and second captured digital images.

In some embodiments, the pixel data of the first digital image isanalyzed to determine how many people are depicted in the first digitalimage, to determine the approximate age of at least one person depictedin the first digital image, or to determine at least one landmarkdepicted in the first digital image, as was described earlier inreference to analyze image step 405.

In some embodiments, a wireless interface, such as wireless modem 350,receives ambient condition information over a wireless network from aprovider, such as a weather service provider. The digital processor 320in the camera phone 300 uses the received ambient condition informationwhen selecting the second possible image capture location, as wasdescribed earlier. The ambient condition information can include, forexample, weather information, geographic location information and timeof day information.

FIG. 4A and FIG. 4B depict two different examples of guidance for imagecapture at different locations based on an analysis of the previousimage received from the user of one of the camera phones 300A (FIG. 4A)and 300B (FIG. 4B). In both examples, the initial guidance provided bythe service provider 280 and transmitted over the communications network250 to the camera phones 300A, 300B is to take a picture of your team onthe railing, as depicted in the initial guidance portion 612 of the userinterface display screen 610 in FIG. 4A and the initial guidance portion622 of the user interface display screen 620 in FIG. 4B.

In the example of FIG. 4A, the image received from the user of cameraphone 300A is a first user captured picture 614 depicting four children.In the example of FIG. 4B, the image received from the user of cameraphone 300B is a second user captured picture 624 of an older couple. Insome embodiments, analyze image step 405 determines the number of peoplein the captured digital images, and the approximate age of one or morethe individuals in the captured digital images, and stores suchdeterminations as metadata in the customer database 288.

Subsequently, in determine next location step 420, the processor 292 inthe computer system 286, or the digital processor 320 in the cameraphone 300, uses these image specific determinations to automaticallyselect the most appropriate next location. In the case of the childrenin the example of FIG. 4A, the Jungle Gym was selected as the nextlocation and appropriate guidance for the Jungle Gym location isdisplayed in next location message area 616 of the user interfacedisplay screen 610 in FIG. 4A. In the case of the older couple in theexample of FIG. 4B, the Park Bench was selected as the next location,and appropriate guidance for the Park Bench location is displayed innext location message area 626 of the user interface display screen 620in FIG. 4B. In both examples, the next location was selected based oninformation determined from analyzing the pixel data of the receivedimage.

FIG. 5 is a high level flow diagram depicting steps for generating photoproducts from images captured at different locations. The photo productsare produced using captured images received from users of camera phone300, along with prestored content provided by service provider 280. Insome embodiments, the prestored content is selected responsive toinformation determined by analysis of the pixel data of the capturedimages received from user of camera phone 300.

In receive image set step 500, an image set including some, or all, ofthe images captured by the user during the experience described inrelation to FIG. 3 is received, for example by retrieving the set ofimages from the customer database 288. In some embodiments, any textentered by a user, such as a team name or the names of the participants,is also retrieved. In some embodiments, geographic location metadata orother information collected or determined during the experience isretrieved, as well as any known information about the customer, and anydata, images, or information returned to the user during the experience.

In evaluate image set step 505, the digital processor 292 in thecomputer system 286, or the processor 320 in the camera phone 300,analyzes the images taken at different locations, which were retrievedin receive image set step 500, according to predetermined criteria andselects images meeting such criteria. In some embodiments, apredetermined number of images is selected, and at least one of theselected images relates to each of a plurality of different locations.

In some embodiments, evaluate image set step 505 analyzes previouslydetermined metadata associated with images that was determined inanalyze image step 405 of FIG. 3. For example, the metadata associatedwith one or more images of the image set can be evaluated to determinewhether the image includes a particular landmark. In this example, thepredetermined criteria can relate to whether the analyzed image includesthe particular landmark.

In some embodiments, evaluate image set step 505 analyzes metadataassociated with the captured digital images received in receive imagestep 400 of FIG. 3. In this example, the predetermined criteria canrelate to whether the analyzed image was captured within a predeterminedarea.

In some embodiments, evaluate image set step 505 analyzes both metadataassociated with the captured digital images received in receive imagestep 400 of FIG. 4 and previously determined metadata associated withimages that was determined in analyze image step 405 of FIG. 3. In thisexample, the predetermined criteria can relate to whether the analyzedimage was captured within a predetermined area and also includes theparticular type of object (e.g. an image of a child, a certain colorautomobile, or a certain type of signpost).

In some embodiments, evaluate image set step 505 includes performingadditional analysis on the pixel data of the received image set inreceive image set step 500, in order to determine the relationshipsbetween images in the image set, or the consistency or quality of theimages in the image set. For example, the image set can be evaluated toselect a subset of images which contain the best composition, or providethe best exposed or focused images. As another example, the image setcan be evaluated to select a subset of images which provide a consistentnumber of individuals in each image, or consistently feature the bestpose (e.g. the best looking smile) of a particular person, such as aparticular child.

In retrieve prestored content step 510, prestored information isretrieved from the custom content database 290. If the photo product tobe produced is a printed photo product, such as a photo booklet, theprestored information can include images, graphics, text, or templates.If the photo product to be produced is a digital photo product, such asslide show or digital video clip, the prestored information can includeaudio information such as voice narration tracks or music tracks, orvideo information such as video clips describing a historic site, orvideo special effects templates or tracks.

In produce photo product step 520, a photo product is produced whichincludes the images selected in evaluate image set step 505 and theprestored content retrieved in retrieve prestored content step 510. Thephoto products that can be produced include, for example, printed pages,photo books, mugs, t-shirts, DVDs, social networking content, digitalslide shows, or other products which utilize the captured images and theretrieved prestored information.

In some embodiments, the selected images from evaluate image set 505 arepositioned in the photo product in association with prestoredinformation that relates to the respective scenes depicted in theselected images, which were captured in a plurality of locationsaccording to predetermined criteria.

In some embodiments, one or more of the images selected in evaluateimage set step 505 can be modified and composited with prestoredinformation. For example, the processor 292 in the computer system 286can process the received captured image in order to crop out the face,convert the face from a color to a monochrome image, and composite theimage of the face into a prestored newspaper template so that theselected image appears to be a photograph in a historic newspaperrelated to a historic site which serves as the theme of the experience.

FIG. 6A-6C depict pages of a first photo product which includes selectedimages positioned in the photo product in association with prestoredinformation. FIG. 6A depicts the first page 800 of the first photoproduct, which is a photo booklet. Page 800 is a cover page, andincludes three images, 802A, 804A, and 806A, which were selectedaccording to predetermined criteria. Images 802A, 804A, and 806A werecaptured by a particular user of a particular camera phone 300. Images802A, 804A, and 806A were captured near the High Falls historic site inRochester, N.Y. on May 7, 2011.

Images 802A, 804A, and 806A are positioned on page 800 in associationwith prestored information, such as a stored panoramic image 812 of theHigh Falls. The stored panoramic image 812 is one example ofpredetermined information that relates to the scene in the user capturedimage 802A, which was captured on the bridge overlooking the High Falls.Page 800 also includes a graphic drawing 810 which depicts the HighFalls area approximately 200 years ago, and graphic drawing 814 thatdepicts a particular event (Sam Patch's jump from the High Falls). Thegraphic drawings 810 and 814 are also examples of predeterminedinformation that relates to the scene in the user captured image 802A.Page 800 also includes a text message 820, “High Falls, Rochester, NewYork”, which provides a title for the first page 800 of the photoproduct. The text 820 is also an example of predetermined informationthat relates to the scene in the user captured image 802A.

Page 800 also includes text 822 which personalizes the first page of thephoto product 800 with the name of the participants “Paul and Brian”depicted in the images 802A, 804A, and 806A. The names can be determinedfrom a text message received from the camera phone 300 in response toearlier guidance provided to the user of the camera phone 300. Page 800also includes text 824 which personalizes the first page of the photoproduct 800 with the date on which the photo experience took place. Thedate can be determined from a real-time clock provided by the computersystem 286 or by date information provided by the camera phone 300 aspart of an MMS message which includes one of the captured images.

FIG. 6B depicts a second page 830 of the first photo product. Page 830includes an image 804B which is a larger sized version of the image 804Athat was included on the first page 800 in FIG. 6A. Page 830 alsoincludes prestored text information 832 which describes the first millthat was built in the High Falls area. Page 830 also includes aprestored image 834, which depicts a plaque located near the millstone.

The image 804B was captured in front of a particular object, which is amillstone, in response to guidance provided to the user of the cameraphone 300. In this example, the guidance to the user was to take apicture of their group near “a circle that came from an Angle”. Theimage of the plaque depicted in prestored image 834 shows that themillstone was donated by Ms. Elizabeth Angle of Irondequoit, N.Y.

FIG. 6C depicts a third page 840 of the first photo product. Page 840includes an image 806B which is a larger sized version the image 806Athat was included on the first page 800 in FIG. 6A. Page 840 alsoincludes prestored text information 844 which describes the Center atHigh Falls area. Page 840 also includes a prestored graphic 842, whichprovides a title and logo related to the location of image 806B, whichwas captured in front of the Center at High Falls, in the High FallsHeritage Area, a major tourist attraction in Rochester, N.Y.

The image 806B on page 840 was captured in response to guidance providedto the user of the camera phone 300. In this example, the guidance tothe user was based on analyzing the pixel data of a previously capturedimage and the ambient conditions. The location was selected from aplurality of possible locations based on the time of day (whichindicated that the Center at High Falls was currently open for visitors,and the number and approximate age of the individuals depicted in theprevious captured images (802A and 804A).

The third page 840 also includes a machine readable code 846 and a humanreadable URL 848. In the specific example shown in FIG. 6C, the machinereadable code 846 is the well-known QR (Quick Response) code, which isreadable by many camera phones. The code consists of modules which arearranged in a square pattern on a white background. In this example, theinformation encoded in the QR code is a link to a website which providesadditional information about the Center at High Falls, and the humanreadable URL 848 provides the same link as plain text. It will beunderstood that QR codes could also be used to provide electronic accessto other images and information, such as the image file associated withthe captured image 806B. The machine readable code 846 and the humanreadable URL 848 are also examples of predetermined information thatrelates to the scene in the user captured image 806B.

FIG. 7A-7C depict pages of a second photo product which includesselected images positioned in the photo product in association withprestored information. In this example, the photo product is provided asa multi-page digital document, such as a pdf file, which is provided tothe user of the camera phone 300, for viewing and possible printing onthe user's home printer. In other embodiments, the photo product can beprinted at a retail establishment for pick-up by the user after theservice provider 280 transmits information to the camera phone 300related to where the printed booklet can be obtained. For example, theinformation can provide the name of a store or a map showing a route tothe store, where the user of the camera phone 300 can pick up their“free photo booklet”.

FIG. 7A depicts the first page 850 and the last page 860 of the secondphoto product. Page 850 is a cover page and includes one captured image,852A, which was captured by a particular user of a particular cameraphone 300 and selected according to predetermined criteria. In thisexample, the captured image 852A was captured by a parent of the childdepicted in the image, in response to guidance which described a firstlocation for capturing an image. The guidance was provided by anautomated phone message from the “Easter Bunny” in response to a textmessage sent by the user of the camera phone 300 to a particular addressspecified on a sign in the North Point Shopping Mall. The phone messageprovided guidance to the parent and child to look for a particularcolored Easter egg in a nearby area of the mall. The guidance furtherasked the parent to photograph their child in front of the Easter eggand to send an MMS message, including the photograph and the child'sname, to a particular address.

In response to the guidance, the parent captured the requested image andtransmitted the image file as part of an MMS message to the serviceprovider. The text message included the child's name, “Henry” along withthe image 852A. The received image was analyzed relative topredetermined criteria to determine if the image included both the faceof a child and a portion of the particular colored Easter egg. Since thecaptured image 852A met the predetermined criteria, it was included onfirst page 850, along with prestored information including the text“Easter Bunny Special Edition” 854 and related graphics. The first page850 also includes prestored graphics information 858 describing thelocation of the egg hunt (e.g. North Point Mall) and a title 856 “TheGREAT HENRY Easter Egg Hunter” which includes the name of the child,“Henry”, included in the text message.

FIG. 7A also depicts a last page 860 of the second photo product. Thelast page 860 includes an advertisement 862 for Kodak Photo Books andKodak Photo Mugs which uses one of the captured digital imagespositioned with other prestored information. For example, the last page860 depicts a photo book 864 which includes a captured image 852B thatis a different sized and cropped version of the captured image 852A onthe first page 850. The last page 860 also depicts a photo mug 868 whichincludes a captured image 852C that is a different sized and croppedversion of the captured image 852A on the first page 850. Both the photobook 864 and photo mug 868 are examples of advertising informationrelated to another product (e.g. a photo book or photo mug) which use atleast one of the images captured using the camera phone 300 and also useprestored information to depict the product offering.

FIG. 7B depicts a second page 870 of the second photo product. Thesecond page 870 includes objects 872A and 872B. Each of the objects 872Aand 872B is intended to be cut out and glued together at the ends inorder to form an Easter egg holder. Portions of the objects 872A and872B can be colored by the child. The objects 872A and 872B include oneof the captured images 874A which is positioned in the Easter egg holderphoto product with prestored information, including graphic linedrawings of Easter Eggs and other items which can be colored by thechild. Second page 870 also includes prestored advertising information876 related to the purchase of glue or other school supplies at aspecific merchant (e.g. Target). In some embodiments, the prestoredadvertising information is selected based on the location of the user,so as to provide the name and location of a nearby merchant which offerssupplies (e.g. glue or crayons) needed to properly complete the photoproduct.

FIG. 7C depicts third page 880 of the second photo product. The thirdpage 880 includes three advertisements in the form of a first coupon882, which provides a discount on ice cream cones, a second coupon 884,which provides a “cash equivalent” discount related to a sandwichmerchant, and third coupon 886, which provides a discount related to apizza merchant. The first coupon 882 includes a differently sized andcropped version of one of the captured images 874B which is positionedwithin the first coupon 882 along with prestored advertising relatedinformation. The second coupon 884 also includes a differently sized andcropped version of one of the captured images 874B which is positionedwithin the second coupon 884 along with other prestored advertisingrelated information. In some embodiments, the prestored advertisinginformation used for coupons 882, 884, and 886 is selected responsive tothe number of persons depicted in the captured image or the approximateage of one or more of the persons depicted in the captured image, orresponsive to the other metadata described earlier in reference to FIG.3.

FIG. 8A and FIG. 8B depict examples of two different photo productswhich utilize images captured by two different users at the samelocation. The particular photo product provided to the two differentusers is determined based on analysis of the pixel data of capturedimages submitted by each of the two different users.

FIG. 8A depicts a first photo product 700, which can be a printed pageor a composite digital image file that can be displayed on the colordisplay 332 of the camera phone 300. The first photo product 700includes a first user image 720 which was received from the user of afirst camera phone 300A. First user image 720 depicts a young couple infront of the Eiffel Tower.

FIG. 8B depicts a second photo product 702 which can also be a printedpage or a composite digital image file. The second photo product 702includes a second user image 722 which was received from the user of asecond camera phone 300B. Second user image 722 depicts three childrenin front of the Eiffel Tower. In both examples, the evaluate image setstep 505 described in reference to FIG. 5 would have determined thelocation as being Paris and determined the numbers of people andapproximate ages of the people in each photo.

The first prestored image content specific page title 710 for the firstphoto product 700 is appropriate to the content of the first user image720 since “The Romance of Paris” likely reflects the young couple'sexperience. The second prestored image content specific page title 712for the second photo product 702 is appropriate to the content of thesecond user image 722, since the children are more apt to view theEiffel Tower as “The Paris Jungle Gym.”

A first image content specific coupon 730, which offers a coupon for “1free bottle of wine at Le Bistro”, is appropriate to the content of thefirst user image 720, which was received from the user of camera phone300A. A second image content specific coupon 732, which provides anoffer of “buy 2 water bottles get 1 free at Le Gift shop” is appropriateto the content of the second user image 722, which was received from theuser of camera phone 300B. The young couple is more likely to want toshare a bottle of wine at a bistro, while a family with young childrenwill be more inclined to get water and souvenirs in the gift shop. Inaddition, making an offer for a discount of a third item, when twosimilar items are purchased, is likely a more appropriate offer thefamily with three small children.

It will be understood, however, that in some embodiments, the selectionof the prestored information, such as prestored advertising, used in theexamples described in relation to FIG. 8A and FIG. 8B might beresponsive to other factors. The factors can include preferenceinformation derived from explicit user input or past behavior, andadditional analysis of the pixel data of the captured digital images,for example, to determine the expressions or demeanor of one or morepeople depicted in the set of images that are evaluated in the evaluateimage set step 505 in FIG. 5. For example, the preference informationfor the user of camera phone 300A might indicate that the user does notdrink alcohol. This could be determined by explicit user input providedat an earlier time or by storing the user behavior to previous offersand determining that the user has never taken advantage of an alcoholrelated offer in the past. In this situation, an offer for a discountrelated to bottled water might be more appropriate, even though analysisof the pixel data of the captured digital image has determined that theimage includes young adults.

As a second example, the expressions or demeanor of the three childrendepicted in the images in FIG. 8B could be determined by analyzing thepixel data of the captured digital images. If such analysis indicatesthat the children have been in a disgruntled mood for an extended periodof time, an offer that provides a discount on a glass of wine (orpossibly a bottle of wine) at a nearby establishment might be welcomedby a parent or guardian who has spent a long afternoon taking photos ofthe three disgruntled children, using camera phone 300B.

In the foregoing detailed description, the method and apparatus of thepresent invention have been described with reference to specificexemplary embodiments thereof. It will, however, be evident that variousmodifications and changes can be made thereto without departing from thebroader spirit and scope of the present invention. The presentspecification and figures are accordingly to be regarded as illustrativerather than restrictive.

A computer program product can include one or more storage medium, forexample; magnetic storage media such as magnetic disk (such as a floppydisk) or magnetic tape; optical storage media such as optical disk,optical tape, or machine readable bar code; solid-state electronicstorage devices such as random access memory (RAM), or read-only memory(ROM); or any other physical device or media employed to store acomputer program having instructions for controlling one or morecomputers to practice the method according to the present invention.

The invention has been described in detail with particular reference tocertain preferred embodiments thereof, but it will be understood thatvariations and modifications can be effected within the spirit and scopeof the invention.

PARTS LIST

-   214 System-   218 Customer Computer-   220 Communication Services Provider (CSP)-   222 Communication Services Provider (CSP)-   224 Kiosk Printer-   240 Cellular Provider Network-   250 Communication Network-   268 Transportation Vehicle-   270 Fulfillment Provider-   272 Web Server-   274 Commerce Manager-   275 Fulfillment Manager-   276 Fulfillment Manager-   278 Digital Printer-   279 DVD Writer-   280 Service Provider-   282 Web Server at Service Provider-   284 Account Manager-   286 Computer System-   288 Customer Database-   290 Custom Content Database-   292 Processor-   300A Camera phone at location A-   300B Camera phone at location B-   300 Camera phone-   302 Flash-   304 Lens-   310 CMOS Image Sensor-   312 Timing Generator-   314 Image Sensor Array-   316 A/D Converter-   318 DRAM Buffer Memory-   320 Digital Processor-   322 RAM-   324 Real Time Clock-   328 Firmware Memory-   330 Image/Data Memory-   332 Color Display-   334 User Controls-   340 Audio Codec-   342 Microphone-   344 Speaker-   350 Wireless Modem-   352 RF Channel-   360 GPS Receiver-   362 Dock Interface-   364 Dock Recharger-   400 Receive Image-   405 Analyze Image-   410 Provide Possible Locations-   415 Final Location Test-   420 Determine Next Location-   430 Provide Guidance-   440 Create Photo Product-   500 Receive Image Set-   505 Evaluate Image Set-   510 Retrieve Prestored Content-   520 Produce Photo Product-   610 User Interface Display Screen-   612 Initial Guidance Portion-   614 First User Captured Picture-   616 Next Location Message Area-   620 User Interface Display Screen-   622 Initial Guidance Portion-   624 Second User Captured Picture-   626 Next Location Message Area-   700 First Photo Product-   710 First Prestored Image Content Specific Page Title-   712 Second Prestored Image Content Specific Page Title-   702 Second Photo Product-   720 First User Image-   722 Second User Image-   730 First Image Content Specific Coupon-   732 Second Image Content Specific Coupon-   800 First Page-   802A User Captured Image-   804A Image-   804B Image-   806A Image-   806B Image-   810 Graphic Drawing-   812 Panoramic Image-   814 Graphic Drawing-   820 Text Message-   822 Text-   824 Text-   830 Second Page-   832 Prestored Text Information-   834 Prestored Image-   840 Third Page-   842 Prestored Graphic-   844 Prestored Text Information-   846 Machine Readable Code-   848 Human Readable URL-   850 First Page-   852A Captured Image-   852B Captured Image-   852C Captured Image-   854 Text-   856 Title-   858 Prestored Graphics Information-   860 Last Page-   862 Advertisement-   864 Photo Book-   868 Photo Mug-   870 Second Page-   872A Object-   872B Object-   874A Captured Image-   874B Captured Image-   876 Prestored Advertising Information-   880 Third Page-   882 First Coupon-   884 Second Coupon-   886 Third Coupon

The invention claimed is:
 1. An image capture device comprising: animage sensor configured to capture a digital image of a scene; an imagedisplay; and a data processing system configured to: display, on theimage display, guidance information for capturing a first digital imageat a location of a first scene; capture, via the image sensor, the firstdigital image of the first scene using the guidance information; analyzepixel data of the first digital image to determine a location of asecond scene and guidance information for locating the second scene,wherein the location of the second scene is selected from a plurality ofpossible locations, and wherein the location of the second scene isdifferent from the location of the first scene; and display, on theimage display, the guidance information for locating the second scene.2. The image capture device of claim 1, wherein the data processingsystem is further configured to provide guidance concerning a locationof a third scene different from the first and second scenes responsiveto an input rejecting the location of the second scene.
 3. The imagecapture device of claim 1, further comprising a wireless interfaceconfigured to transmit the first digital image to a service provider. 4.The image capture device of claim 1, wherein the data processing systemis further configured to analyze the pixel data of the first digitalimage to determine how many people are depicted in the first digitalimage.
 5. The image capture device of claim 1, wherein the dataprocessing system is further configured to analyze the pixel data of thefirst digital image to determine an approximate age of a person depictedin the first digital image.
 6. The image capture device of claim 1,further comprising a wireless interface configured to receive ambientcondition information, wherein the data processing system is furtherconfigured to determine the location of the second scene based on theambient condition information.
 7. The image capture device of claim 6,wherein the ambient condition information includes weather information.8. The image capture device of claim 6, wherein the ambient conditioninformation includes geolocation information.
 9. The image capturedevice of claim 6, wherein the ambient condition information includestime of day information.
 10. The image capture device of claim 1,wherein the data processing system is further configured to analyze thepixel data of the first digital image to determine a landmark depictedin the first digital image.
 11. The image capture device of claim 1,wherein the data processing system is further configured to cause theimage sensor to capture a second digital image of the second scene basedon the second image guidance information.
 12. A method comprising:displaying, on an image display of a computing device, guidanceinformation for capturing a first digital image at a location of a firstscene; capturing, by the computing device, the first digital image ofthe first scene using the guidance information; analyzing, by thecomputing device, pixel data of the first digital image to determine alocation of a second scene and guidance information for locating thesecond scene, wherein the location of the second scene is selected froma plurality of possible locations, and wherein the location of thesecond scene is different from the location of the first scene; anddisplaying, by the computing device, the guidance information forlocating the second scene.
 13. The method of claim 12, furthercomprising providing guidance concerning a location of a third scenedifferent from the first and second scenes responsive to an inputrejecting the location of the second scene.
 14. The method of claim 12,wherein the analyzing the pixel data of the first digital imagecomprises analyzing the pixel data of the first digital image todetermine how many people are depicted in the first digital image. 15.The method of claim 12, wherein the analyzing the pixel data of thefirst digital image comprises analyzing the pixel data of the firstdigital image to determine an approximate age of a person depicted inthe first digital image.
 16. The method of claim 12, further comprisingreceiving ambient condition information; and determining the location ofthe second scene based on the ambient condition information.
 17. Themethod of claim 16, wherein the ambient condition information comprisesat least one of weather information at the location of the second scene,geolocation information of the first scene or second scene, or time ofday information.
 18. The method of claim 12, wherein the analyzing thepixel data of the first digital image comprises analyzing the pixel dataof the first digital image to determine a landmark depicted in the firstdigital image.
 19. The method of claim 12, further comprising capturinga second digital image of the second scene based on the second imageguidance information.
 20. A non-transitory computer-readable mediumhaving instructions stored thereon that, upon execution by a computingdevice, cause the computing device to perform operations comprising:displaying guidance information for capturing a first digital image at alocation of a first scene; capturing the first digital image of thefirst scene using the guidance information; analyzing pixel data of thefirst digital image to determine a location of a second scene andguidance information for locating the second scene, wherein the locationof the second scene is selected from a plurality of possible locations,and wherein the location of the second scene is different from thelocation of the first scene; and displaying the guidance information forlocating the second scene.